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SECTION  I 

INSTRUCTIONAL  SYSTEMS  DEVELOPMENT: 
A CONCEPTUAL  ANALYSIS 


Background 

In  late  1974,  an  attempt  was  undertaken  to  define  the  state-of-the-art 
of  the  Systems  Approach  to  Training  (SAT).  This  task,  which  was  originally 
envisioned  as  a straightforward  literature  review,  was  to  serve  as  the  design 
basis  for  a research  program  aimed  at  expanding  available  SAT  technology. 
However,  preliminary  work  indicated  that  the  size  and  complexity  of  the  rele- 
vant lite*'ature,  and  therefore  the  scope  of  the  planned  project  were  consid- 
erably larger  than  anticipated.  It  vias  found  for  example,  that  o\'er  100  SAT 
manuals  had  been  published  between  1960  and  1975.  The  problem  of  analyzing 
the  literature  was  exacerbated  by  an  abundance  of  idiosyncratic,  loosely 
defined  terminology  which  proved  to  be  the  source  of  a great  deal  of  confu- 
sion. SAT,  for  instance,  is  referred  to  by  a variety  of  alternative  appella- 
tions including:  the  Systems  Engineering  of  Training  (SET),  Training  Situa- 

tion Analysis  ("^SA),  the  Developmental  Approach  to  Training  (DAT),  the  Design 
of  Instructional  Systems  (DIS),  and  most  recently,  ItiStructional  Systems  De- 
velopment (ISD). 

Differences  among  available  formulations  of  the  SAT  concept 
range  from  superficial  '*>riations  in  terminology  to  fundamental  variations  in 
philosophy.  The  basic  issue  on  which  disagreement  exists  is  the  degree  to 
which  the  instructional  design  process  can  be  reduced  to  a linear  sequence  of 
generally  applicable,  prescriptive  procedures.  The  positions  that  have  been 
espoused  range  from  Eckstrand's  (1964)  statement  that  the  design  of  instruc- 
tion is  primarily  an  art,  to  the  hypothesis  that  course  design  can  be  reduced 
to  a series  of  well  defined  procedures  which  can  be  carried  out  by  untrained 
personnel . 

The  size  of  the  SAT  literature,  the  complexity  of  the  concept,  and  the 
controversy  which  surrounds  it,  serve  to  emphasize  the  need  for  adequately 
assessing  the  state-of-the-art  of  SAT  before  embarking  on  further  attempts  to 
expand  it.  Heeding  Santayana's  warning  that  those  who  are  not  aware  of  the 
past  a*"e  condemned  to  reoeat  ■>t,  this  bibl iog'*aphy  was  co5''p'fled  as  a first 
step  in  assessing  the  state-of-the-art.  The  bibliography  was  designed  to 
allow  researchers,  developers  and  appliers  of  SAT  to  enhance  the  effectiveness 
of  their  future  efforts  by  taking  greater  advantage  of  the  work  that  has 
already  been  accomplished. 

The  Proolem  of  Definition 


The  central  issue  in  selecting  the  content  and  organization  of  the  bibli- 
ogrdpny  was  to  define  the  "systems  approach  to  training"  or  equivalently 
"instructional  systems  development."  Of  the  dozens  of  definitions  contained 
in  the  SAT  manuals  and  in  the  related  literature,  none  have  attained  wide- 
spread acceptance.  Campbell  (1971)  points  out  that  a major  problem  with  the 
available  definitions  is  that  they  do  not  indicate  how  their  particular 
methodology  differs  from  other  S.AT  methodologies  or  from  traditional  ways  of 
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developing  training  programs.  The  present  state  of  affairs  is  characterized 
by  general  acceptance  of  the  terms  "SAT"  and  "ISD"  together  with  widespread 
disagreement  as  to  what  the  terms  mean.  This  confusion  over  the  nature  of 
SAT  is  primarily  a result  of  three  factors:  lack  of  terminological  standardi- 

zation, problems  associated  with  educational  innovations,  and  the  evolution- 
ary nature  of  the  SAT  concept.  Each  of  these  will  now  be  discussed  in  detail. 

{:)  Lack  of  terminological  standardization 

Analysis  of  similarities  among  existing  SAT  manuals  indicates  a high 
degree  of  overlap  in  the  terminology  used.  Almost  all  of  the  manuals  use  tne 
terms:  task  analysis,  behavioral  objectives,  methodology,  media  selection, 

sequencing,  objective  performance  measurement,  criterion  referenced  testing, 
individualized  instruction,  and  quality  control.  The  use  of  a common  termi- 
nology creates  the  initial  impression  of  high  content  similarity  among  the 
manuals.  However,  closer  inspection  of  the  operational  definitions  given  to 
these  terms  shows  this  impression  to  be  mistaken.  The  following  example  is 
illustrative. 

Virtually,  all  of  the  SAT  manuals  use  the  term  "task  analysis  but 
the  operational  definitions  of  task  analysis  provided  by  the  manuals  differ 
both  in  content  and  in  degree  of  detail.  While  some  manuals  leave  much  to 
the  discretion  of  the  analysts,  others  are  so  specific  as  to  provide  a form 
which  need  only  be  completed  by  the  analyst.  Some  require  all  skills  to  be 
broken  down  into  hierarchical  categories  such  as  role,  duty  and  activity,  or 
job,  task,  and  element;  others  provide  little  or  no  structure  as  to  the  num- 
ber or  the  types  of  categories  to  be  used.  Some  require  each  task  to  be 
classified  as  psychomotor,  cognitive  or  affective;  others  rate  each  task  on 
each  of  these  categories;  still  others  ignore  this  breakdown.  The  task  infor- 
mation called  for  varies  among  manuals  but  usually  includes  some  of  the  fol- 
lowing: criticality,  frequency,  initiation  and  completion  cues,  degree  of 

judgment  required,  preceding  and  subsequent  tasks,  etc. 

SAT  manuals  generally  provide  only  one  procedure  for  task  analysis. 
When  alternative  procedures  are  not  provided,  the  assumption  is  made  that  the 
method  given  is  universally  applicable.  This  assumption  is  not  warranted  by 
the  literature  on  task  analysis.  The  fiilhrefhs,  in  their  pioneering,  turn 
of-the-century  work  on  improving  industrial  efficiency,  developed  the  first 
formalized  task  analytic  methodology.  Their  procedures  were  useful  in  time 
and  motion  studies  on  production  line  tasks.  However,  in  the  1950's,  R.  B. 
Milder  found  that  the  Gilbreths'  procedures  did  not  allow  for  the  identifica- 
tion of  human  attributes  used  in  complex  tasks  (Swain,  1962).  Miller  devel- 
oped a methodology  entitled  "task-demands  analysis"  because  he  believed  that 
existing  methods  did  not  provide  adequate  data  concerning  the  demands  which 
tasks  make  on  the  operator.  Since  1960,  the  number  of  available  task  ana- 
lytic methodologies  has  risen  dramatically.  A number  of  theorists,  after 
reviewing  this  state  of  affairs,  have  concluded  that  no  single  method  of  task 
analysis  can  be  generated  which  is  valid  in  all  circumstances  (Gustafson, 
Honsberger,  and  Michelson,  1960;  Folley,  1964;  DeGreen,  1970;  Rankin,  1974). 

The  degree  to  which  tasK  analysis  can  or  should  be  procedural ized  is 
controversial.  The  trade-off  is  that  although  higher  degrees  of  procedurali- 
zation  result  in  narrower  ranges  of  application,  they  may  permit  the  use  of 
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less  qualified,  less  costly  analysts.  DeGreen's  (1970)  analysis  of  this 
problem  led  him  to  conclude  that:  (a)  reduction  of  task  analysis  to  a 
routine  checklist  procedure  results  in  "a  deluge  of  useless  data";  (b)  task 
analysis  must  always  be  viewed  as  a means  and  not  as  an  end;  and  (c)  the 
usefulness  of  task  analytic  data  is  a function  of  the  degree  of  expertise 
of  the  analyst. 


Most  SAT  manuals  have  failed  to  inform  the  user:  of  the  controversies 
that  have  been  described  above,  of  the  existence  of  alternative  methodolo- 
gies, of  the  need  to  modify  given  techniques  to  fit  specific  circumstances, 
or  of  the  experience,  training  and  skills  necessary  to  perform  valid  task 
analysis.  The  failure  to  provide  this  information  has  prevented  users  from 
the  benefit  of  the  experience  of  others  as  described  in  the  literature,  and 
may  have  caused  them  to  acquire  a naively  simple  understanding  of  task 
analysis. 


Although  the  example  used  here  is  "task  analysis,"  a similar  presenta- 
tion could  be  made  concerning  each  of  the  terms  shared  by  the  majority  of 
SAT  manuals:  behavioral  objectives,  media  selection,  methodology  selection, 
criterion  referenced  testing,  objective  performance  measurement,  sequencing, 
quality  control,  etc.  Research  and  analysis  are  needed  to  determine  for 
each  of  these  concepts,  the  degree  to  which  procedural ization  can  be 
achieved,  the  generality  of  those  procedures,  and  the  skills  necessary  to 
apply  them. 

(2)  The  problems  of  educational  innovations 


The  secona  factor  contributing  to  the  confusion  concerning  the 
nature  of  SAT  has  been  its  '^mergence  as  an  educational  innovation.  Students 
of  the  history  of  education  have  long  been  aware  that  the  courses  which  edu- 
cational innovations  take  are  shaped  by  factors  other  than  their  inherent 
aavantages  and  limitations.  Camobell  (1971)  stated  that  educational  innova- 
tions have  historically  followed  a predictable  life  cycle,  and  constructed  a 
three-stage  model  of  that  process.  In  the  firiic  stage,  a new  technique 
appears  and  develops  a large  following  of  advocates  who  claim  to  have  suc- 
cessfully applied  the  technique.  The  second  stage  consists  of  numerous  modi- 
fications of  the  basic  technique.  The  third  and  final  stage  in  the  life 
cycle  of  educational  innovations  is  the  appearance  of  criticism  by  a few 
vocal  opponent;,  which  grov;s  into  an  inevitable  baukldsh.  According 
model,  this  criticism  does  not  serve  to  stimulate  improvement  of  the 
nique,  but  to  stimulate  the  development  of  a new  technique.  At  that 
the  cycle  starts  anew. 


to  the 
tech- 
time, 


Although  Campbell's  model  is  primarily  descriptive,  Milsum  (1968)  pre- 
sents a phenomenom  called  the  "bandwagon  effect"  which  helps  explain  the 
model.  The  bandwagon  effect  serves  to  transform  researchable  hypotheses 
(educational  innovations)  into  political  entities,  thereby  triggering  the 
mechanism  which  leads  to  the  innovation's  downfall.  The  mechanism  works  as 
follows.  As  the  number  of  researchers,  developers,  theorists,  administra- 
tors, laboratories,  schools,  etc.,  who  have  vested  interest  in  the  innovation 
grows,  the  resistance  to  critical  examination  of  the  innovation  and  to  the 
consideration  of  alternatives  also  grows.  In  addition,  claims  are  made  for 
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the  innovation  by  those  with  vested  interest  which  are  unreasonably  optimis- 
tic. In  this  way,  the  innovation  attains  the  reputation  as  a "panacea", 
that  is,  as  a widely  applicable  technique  which  promises  extremely  high 
pay-offs  for  relatively  small  inputs.  One  reason  for  this  occurrence  is 
that  timidity  concerning  possible  R&D  pay-offs  is  not  conductive  to  success 
in  the  competition  for  research  and  development  funds.  A second  reason  is 
that  it  is  more  prestigious,  more  conducive  to  advancement,  and  more  fun  to 
be  associated  with  the  development  of  a highly  visible  technique  which  has 
the  possibility  of  revolutionizing  the  educational  community. 

The  process  by  which  the  innovation  attains  the  reputation  of  a panacea 
has  an  unwanted  side  effect.  The  greater  the  number  of  people  who  attempt 
to  use  innovation  based  on  unfulfillable  promises,  the  greater  the  number  of 
people  will  be  who  are  aisappointed  by  it.  As  this  number  grows,  the  criti- 
cism and  backlash  predicted  by  Campbell's  model  occur  and  eventually  result 
in  the  downfall  of  the  innovation. 

The  history  of  education  is  replete  with  examples  of  innovations  which 
have  fallen  victim  to  these  problems:  the  teaching  machine,  programmed 

instruction,  adaptive  training,  team  teaching,  microteaching,  accountability, 
the  voucher  system,  behavior  modification  in  the  classroom,  performance 
contracting,  the  "free"  school,  the  "open"  classroom.  Project  Headstart,  and 
others.  According  to  Campbell  (1971),  SAT  is  the  current  innovation,  and  it 
too  is  following  the  life  cycle  predicted  by  his  model. 

The  fact  that  SAT  has  been  touted  as  a panacea  and  has  fallen  victim  to 
the  bandwagon  effect  was  first  documented  in  1968  by  Hartley.  He  concluded 
that  the  SAT  literature  is  "long  on  persuasion  and  short  on  critical  self 
appraisal".  He  believed  this  to  be  the  result  of  ovt-'^zealousness  in  at- 
tempts to  use  the  new  methodology  without  a clear  understanding  of  what  it 
was  supposed  to  produce.  Carter's  1969  article,  "The  Systems  Approach  to 
Education:  Mystique  and  reality"  provides  not  only  a review  of  the  problems 
created  by  the  bandwagon  viffect  but  also  e realistic  assessment  of  what  can 
be  exoected  from  SAT.  Sugarman,  Johnson  and  Hinton  (1975)  and  Montemerlo 
(1975)  provide  further  data  and  analysis  in  these  two  areas. 

Campbell's  model  and  Milsum's  description  of  the  bandwagon  effect  have 
enhanced  our  understanding  of  the  state-of-the-art  of  SAT.  The  studies 
referenced  in  the  preceding  paragraph  have  documented  the  problems  accrued 
by  SAT  which  were  predictable  from  Campbell's  and  Miisum's  work,  namely: 
its  transformation  into  a political  entity,  the  >^esi stance  to  constructive 
critical  assessment  of  SAT,  and  its  attainment  of  an  oversold  reputation. 
These  studies  have  also  indicated  the  courses  of  action  necessary  to  amelio- 
rate the  conditions  caused  by  those  problems.  The  first  is  to  prevent  the 
backlash  predicted  by  Campbell's  model  which  is  caused  by  the  growing  reali- 
zation that  an  innovation  cannot  live  up  to  an  oversold  reputation.  This 
can  only  be  accomplished  by  reducing  those  expectations  to  realistic  pro- 
portions. The  second  course  of  action  is  to  remove  SAT  from  its  status  as  a 
political  entity,  and  thereby  facilitate  constructive  criticism  and  the  con- 
sideration of  alternatives.  In  order  to  do  this,  high-level  SAT  advocates 
must  be  fully  advised  of  both  the  advantages  and  the  limitations  of  the  con- 
cept, and  of  what  has  historically  happened  to  educational  innovations  which 


8 


NAVTRAEQUIPCEN  IH-257 


have  become  entrapped  in  the  political  arena.  The  third  and  final  strategy 
is  to  subject  SAT  to  rigorous  analytical  and  empirical  investigation  in 
order  to  further  delineate  and  validate  its  advantages  and  limitations. 

These  three  courses  of  actio.n  v;ill  minimize  the  political  factors  which  have 
hindered  real  progress  in  advancing  the  state-of-the-art  of  SAT. 

(3)  The  evolutionary  nature  of  the  SAT  process 

The  third  and  final  factor  clouding  the  definitional  issue  has  been 
the  evolutionary  process  which  SAT  has  undergone.  Historical  data  collected 
during  the  compilation  of  this  bibliography,  which  can  aid  in  resolving  this 
issue,  will  now  be  presented. 

(a)  Systems  analysis 

The  systems  approach  to  training,  SAT,  evolved  from  "sys- 
tems analysis"  (alternatively  called  "the  systems  approach")  a methodology 
developed  during  World  War  II,  to  solve  problems  created  by  rapidly  advancing 
weapons  systems  technology.  After  the  war,  the  methodology  was  found  useful 
in  the  solution  of  problems  in  a variety  of  fields.  The  problems  for  which 
systems  analysis  was  found  to  be  app  ‘opriate  are  those  which  are  not  solvable 
using  existing  procedures,  and  whose  complexity  strains  human  comprehension 
when  initially  viewed  in  their  entirety.  Systems  analysis  possesses  three 
main  features  which  make  it  uniquely  powerful  tool  in  solving  such  problems. 
The  first  is  the  use  of  an  interdisciplinary  team  of  experts  to  ensure  that 
as  much  relevant  information  as  possible  is  brought  to  bear  in  solving  the 
problem,  and  that  all  aspects  of  the  problem  are:  (1)  identified,  (2)  con- 

sidered in  terms  of  their  relative  importance,  and  (3)  considered  from  dif- 
ferent points  of  view.  The  second  feature  is  the  use  of  "models",  that  is, 
simplifications  of  the  problem  which  aid  in  initially  understanding  highly 
complex  problems  by  reducing  them  to  analyzable  proportions.  The  third 
feature  is  the  design  of  a unique  method  for  solving  the  problem  which  is  as 
systematic  as  the  proble'i  will  allow.  The  interdisciplinary  team  of  experts, 
which  creates  and  implements  this  design,  retains  the  right  to  replace  or 
modify  it  at  any  point  during  the  analysis. 

Systems  analysis  does  not  necessarily  result  in  the  "best" 
solution  to  a problem.  It  merely  insures  that  the  best  qualified  people  have 
gathered  as  much  relevant  information  as  possible,  and  have  recommended  a 
solution,  which  in  their  judgment,  is  better  than  the  alternatives.  In  short, 
systems  analysis  produces  a "best  educated  guess."  Tiie  alternatives  to  sys- 
tems analysis  in  solving  problems  of  the  type  described  above,  are:  the  use 
of  personnel  with  less  than  the  best  qualifications,  and/or  the  consideration 
of  less  information  than  is  available.  .Since  systems  analysis  is  the  most 
costly  of  the  alternatives,  it  is  employed  only  when  failure  to  solve  the 
problem  involves  sufficient  potential  danger  or  loss  to  warrant  the  added 
expense. 


(b)  Systems  analysis  applied  to  training 

In  the  late  1950's,  the  first  attempts  to  apply  systems 
analysis  to  the  design  of  training  programs  were  undertaken  by  the  Rand 
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Corporation  (Kershaw  and  McKean,  1958;  Kershaw,  1959)  which  was  responsible 
for  much  of  the  development  of  systems  analysis  itself,  and  by  the  Human 
Resources  Research  Organization  (Hoehn,  I960)  The  HumRRO  work  was  summar- 
ized by  Crawford  at  the  Naval  Training  Device  Center's  Seminar  on  Human 
Factors  in  Military  Training  (16  March  1961).  His  presentation  included  a 
f'ow  chart  describing  the  systems  approach  to  training  as  a seven  stage 
process.  This  flow  chart  has  served  as  a prototype,  both  in  format  and  in 
content,  for  those  which  appear  in  virtually  all  of  the  SAT  literature  which 
f ol 1 owed . 

The  early  SAT  literature  drew  heavily  on  the  techniques  of  sysu’.ms 
analysis.  The  design  of  large  scale  training  programs  was  viewed  by  early 
SAT  developers  as  a highly  complex  task  which  could  not  be  accomplished 
procedurally.  SAT  provided  an  alternative  to  the  traditional  approach  to 
training  program  design  which  relied  solely  on  subject  matter  experts. 
Although  the  training  programs  developed  by  such  personnel  are  effective  in 
teaching  the  desired  skills,  they  are  generally  not  as  efficient  as  they 
could  be  if  the  training  program  design  team  had  possessed  expertise  in 
educational  psychology,  training  technology,  and  systems  analysis.  The 
increased  cost  incurred  in  the  use  of  systems  analytic  techniques  by  expr. ‘ 
training  program  designers  will  be  more  than  offset  by  the  increased  eff; 
ciency  of  the  programs  they  produced. 

The  goal  of  early  SAT  developers  was  to  generate  tools  which  could  aid 
training  program  design  experts  in  their  day-to-day  work.  These  tools  con- 
sisted of  models,  that  is.  formalized  simplifications  of  methods  and  tech- 
niques which  other  expe*"ts  '••aJ  found  useful.  These  models  were  intended  to 
be  used,  modified  or  ignored,  in  any  particular  situation,  based  on  the 
discretion  of  the  user.  They  were  not  intended  to  relieve  him  of  his 
responsibility  as  a decision  maker. 

(c)  The  bifurcation 

The  early  1960’s  witnessed  the  emergence  of  a new  technol- 
ogy which  greatly  affected  the  evolution  of  SAT.  This  new  technology  was 
based  on  the  hypothesis  that  if  training  program  design  experts  could  formal- 
ize models  of  the  methods  and  techniques  that  made  them  successful,  then 
•'‘laymen  could  follow  these  models  and  produce  the  same  result  at  lower  cost. 
The  main  thrust  of  developmental  efforts  under  this  technology  has  been  the 
produrHon  of  manuals  v.nich  attempt  to  reduce  the  design  of  training  pro- 
grams to  a linear  sequence  of  procedures  w.iich  can  be  carried  out  by  person- 
nel inexperienced  in  training  program  design. 

This  new  technology  was  quite  different  from  SAT.  According  to  SAT, 
training  program  development  is  a complex  problem  which  cannot  be  solved 
procedurally,  and  therefore  requires  the  techniques  of  systems  analysis. 
According  to  the  new  technology,  training  program  development  can  be  accom- 
plished by  alayman  using  a procedural ized  manual,  thus  rendering  systems 
analysis  inappropriate.  The  new  technology  and  SAT  are  fundamentally  at 
odds  both  in  philosophy  and  in  practice.  A great  deal  of  confusion  was 
caused  when  the  new  technology,  which  held  that  systems  analysis  (i.e.,  the 
systems  approach)  is  inappropriate  to  training  program  development,  adopted: 
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the  name  of  SAT,  its  flow  charts,  and  much  of  its  terminology.  To  ensure 
clarity  during  the  rescainder  of  this  paper,  the  original,  generic  concept  of 
SAT  will  continue  to  be  referred  to  as  SAT,  where  the  new  technology  will  be 
referred  to  as  "procedural i zed  SAT." 

(d)  The  orocedural ized  SAT  manuals 


During  the  mid  and  late  I960's,  the  concept  of  procedur- 
alized  SAT  attained  widespread  popularity  in  both  the  military  and  civilian 
communities.  Over  one  hurdrod  procedural i zed  SAT  manuals  were  published 
between  1960  and  1975,  mostly  by  the  military.  The  Navy  first  entered  this 
field  with  the  development  of  the  Training  Situation  Analysis  (TSA)  method- 
ology (Bertin,  1963;  Van  Albert  et  al , 196A;  Chenzoff  and  Folley,  1965).  The 
Army  began  with  Project  Minerva,  an  Army  Security  Agency  study  which  resulted 
in  the  Design  of  Instructional  Systems  (DIS)  manual  in  1966.  An  e.xcellent 

cd  by  Tracey,  rlymi  and  Legere  (1967).  Two 
period  were  Butler's  (1967)  Instructional 


ever  View  of  thic  pv'oject  ;;rovi 

other  influential  manuals  of  this 


Systems  Development  (ISD)  manual,  wnich  was  written  under  Job  Corps  auspices, 
and  Pvjndquist's  course  design  manual  (1966,  1967,  1970)  which  was  developed 
at  the  Navy  Personnel  Research  and  Development  Center. 


Later  SAT  manuals  developed  by  the  armed  services  include  the  Army's 
CONARC  REG  350-100-1,  Systems  Engineering  of  Training,  (1968);  the  Air  Force's 
Manual  50-2,  Instructional  Systems  Development,  (1970);  and  the  Air  Force's 
AFP  50-58,  Handbook  for  Designers  of  Instructional  Systems,  (1973).  The 
Army,  Navy  and  Air  Force  participated  ’n  the  development  or  an  Interservice 
ISD  manual  which  was  published  late  in  1975. 

These  procedural i zed  SAT  manuals  subdivide  training  p‘'ogram  development 
into  a number  of  linear  steps  which  generally  include:  task  analysis, 

specific  behavioral  objectives,  selection  and  sequencing  of  tasks  for  train- 
ing, media  and  methodology  selection,  development  of  objective  tests,  initial 
course  implementation  and  refinement,  and  quality  control.  It  is  interesting 
Lo  note  that  a methodology,  similar  to  the  procedural i zed  SA"^  manuals  of  the 
1960-1975  era,  was  in  use  by  the  Air  Force  in  the  1940's.  The  Report  of  the 
Training  Analysis  and  Development  Conference  held  at  Scott  AFB  on  22-24  Octo- 
ber 1951  describes  that  methodology,  which  was  called  the  Developmental 
Approach  to  Training  (DAT).  DAT  specifically  entailed-  task  and  traiping 
analyses,  specific  objectives,  sequencing,  objective  measurement,  quality 
control,  and  selection  of  optimal  training  methods.  It  emphasized  the  "mis- 
sion approach  to  training/'  and  the  measurement  of  performance  rather  than  of 
verbalization.  This  methodology  which  predated  SAT  by  over  a decade  was  not 
tied  to  systems  analysis.  The  report  does  not  state  when  DAT  was  developed, 
but  LtCoi  Ferguson  saic  at  the  1951  conference  that  he  had  been  using  it  for 
34  years.  The  report  noted  that  DAT  management  personnel  faced  a problem  in 
1951  which  plagues  their  ISD  counterparts  in  1975,  the  lack  of  uniformity 
with  which  their  methodology  v/as  interpreted  and  applied.  The  DAT  concept, 
while  similar  in  many  respects  to  TSA,  SAT,  ISD,  DIS,  and  SET  was  not  formal- 
ized to  as  great  a degree. 

The  development  of  procedural ized  SAT  manuals  has  continued  from  the 
early  1960's  to  the  present.  However,  little  is  known  ccncerning  the  degree 
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to  w^ich  these  attempts  have  been  successful.  The  only  laroe  scale  empirical 
evaluation  of  such  a manual  was  carried  out  by  the  Human  Resources  Research 
Organization  at  the  request  of  the  Army.  Ricketson,  Shulz  and  Wright  (1970) 
evaluated  the  capability  of  personnel  inexperienced  in  training  program  devel- 
opnent  to  implement  CONARC  REG  350-100-1,  Systems  Engineering  of  Training. 

They  found  that  the  manual  told  the  user  v/hat  to  do,  but  not  how  to  do  it. 

As  a result,  little  use  was  made  of  current  training  technology,  and  the 
users  tended  to  develop  training  courses  resembling  those  with  which  they 
had  been  taught.  The  manual  required  a great  deal  of  paperwork,  19  major 
products  and  81  subproducts.  However,  the  users  often  did  not  know  the  pur- 
pose of  much  of  this  documentation  and,  as  a result,  often  ignored  the  paper- 
work after  completing  it. 

The  general  finding  of  the  evaluation  was  that  the  marual,  when  used  by 
inexperienced  training  program  developers,  did  not  result  in  efficient  train- 
ing orogram;  litilizing  state-of-the-art  training  technology.  In  other  wo*'ds. 
they  found  that  in  the  case  of  the  CONARC  REG  35C-100-1 , the  goal  of  develop- 
ing a model  of  the  expert  course  developer,  which  would  allow  a layman  to 
imitate  his  methods  and  thus  produce  similar  results  at  a lower  cost,  had  not 
been  achieved.  Of  course,  the  results  of  this  investigation  do  not  imply 
that  all  of  the  procedural i zed  SAT  manuals  are  ineffective  or  that  they  can 
not  be  made  effective.  They  do  emphasize,  however,  the  need  to  validate 
future  manuals  before  implementing  them  on  a large  scale.  They  also  indicate 
the  need  for  further  investigation  into  the  question  of  which  portions  of 
training  program  development  lend  themselves  to  procedural ization,  the  degree 
of  procedure lization  which  can  be  realistically  expected,  and  the  degree  to 
which  the  procedures  are  general izable  across  types  of  training  problems. 

(e)  Revival  of  the  original  SAT  concept 


During  the  middle  and  late  1960's,  the  procedural ized  SAT 
concept  generated  a great  deal  of  literature.  The  original,  generic  concept 
of  SAT,  which  remained  relatively  dormant  during  that  period,  has  been  the 
subject  of  renewed  interest  during  the  1970's.  This  is,  at  least  partially, 
a result  of  a re-evaluation  of  the  state-of-the-art  of  educational  psychology 
(Campbell,  197i;  Glaser  and  Resnick,  1972;  McKeacchie,  l9/4),  which  has  con- 
cluded that  the  available  theory  and  empirical  evidence  on  the  process  of 
learning  and  teaching  do  not  support  the  procedurdl ization  of  the  training 
program  development  process.  As  McKeatchie  (1974)  points  out,  psychologists 
are  much  less  sure  of  the  "laws  of  learning"  than  they  were  a few  years  ago. 


Recent  research  under  the  original  SAT  conceptualization,  which  is  in 
agreement  with  these  conclusions,  is  again  attempting  to  develop  methods, 
models,  and  techniques  which  training  experts  can  use,  modify  o>  ignore. 

This  reflects  a re-emergence  of  the  1950's  belief  that  development  of  train- 
ing is  a complex  problem,  not  solvable  by  procedural ized  methods,  but 
requiring  the  techniques  of  systems  analysis.  One  area  in  which  this  type  of 
work  is  being  carried  out  is  that  of  the  selection  of  training  media.  Braby, 
et  al  (1975),  and  Boucher,  Gottleib  and  Morganlander  (1973)  have  produced 
media  selection  models  which  specifically  state  that  their  goal  is  to  assist 
rather  than  to  replace  the  experienced  specialist.  Braby,  et  al  (1975)  state: 
"The  choosing  of  an  optimal  instructional  delivery  system  for  various  types 
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of  military  training  objectives  remains  a subtle  and  complex  decision  making 
task;  something  that  can  not  be  fully  procedural i zed.  Training  systems 
designers  who  use  the  TECEP  technique  must  possess  expert  knowledge  of  media. 
The  technique  will  serve  as  a performance  aid  in  carefully  exploring  the 
probable  cost  and  effectiveness  of  various  alternatives,  including  innova- 
tions". With  this  statement,  they  have  captured  the  essence  of  the  original 
and  the  generic  meaning  of  the  systems  approach  to  training. 

At  present,  both  the  original  and  the  procedural ized  concepts  of  SAT 
are  active.  However,  since  both  use  the  same  terminology,  each  particular 
piece  of  literature  must  be  read  to  determine  the  conceptualization  of  SAT 
under  which  it  falls.  The  SAT  literature  has  been  a prime  source  of  confusion 
concerning  the  nature  of  SAT.  The  history  of  the  SAT  concept,  as  it  comes  to 
light,  should  aid  in  reducing  this  confusion. 
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Summary  and  Conclusions 

The  voluminous  SAT  literature  produced  over  the  past  two  decades 
reveals  an  underlying  confusion  concerning  the  nature  of  SAT.  The  same 
terms  are  used  to  refer  to  different  methodologies,  thereby  yielding  the 
illusion  of  a greater  degree  of  agreement  than  actually  exists.  The  empiri- 
cal studies  needed  to  validate  the  various  methodologies  and  to  evaluate  the 
real  differences  among  them,  have  not  been  accomplished.  Educational  histo- 
rians have  noted  that  this  is  a typical  occurrence  in  the  life-cycle  of  edu- 
cational innovations.  A bandwagon  effect  takes  hold  and  transforms  the  inno- 
vation into  a political  entity,  suppressing  empirical  validations,  construc- 
tive criticism,  and  the  consideration  of  alternatives.  To  further  complicate 
matters,  two  opposing  schools  of  thought  as  to  the  nature  of  SAT  have 
evolved,  coexisted,  and  gone  under  the  same  name.  The  three  issues,  termi- 
nology, political  problems,  and  the  evolution  of  the  concept,  which  have 
caused  much  of  the  existing  confusion,  have  been  identified  and  discussed 
here  as  a first  step  toward  defining  the  state-of-the-art  of  S^T. 

A great  deal  of  research  is  needed  to  further  refine  and  articulate  the 
SAT  concept.  To  be  effective,  it  must  include  empirical  investigations. 
Failure  to  do  so  in  the  past  has  resulted  in  the  development  of  over  one 
hundred  procedural i ■’.ed  SAT  methodologies,  none  of  which  has  been  determined 
to  be  more  effective  than  any  of  the  others.  Empirical  validations  of  SAT 
methodologies  are  expensive,  time  consuming  and  difficult  to  control.  There- 
fore, the  necessary  experimentation  must  be  preceded  by  analysis  to  insure 
that  the  methodologies  to  be  evaluated  are  as  complete,  as  internally  consis- 
tent, as  continuous  with  existing  knowledge,  and  as  potentially  useful  as 
possible.  This  bibliography  was  compiled  to  aid  in  these  analytic  endeavors. 
Through  comprehensive  literature  surveys,  SAT  developers  can  insure  that 
fullest  advantage  has  been  taken  of  existing  technology,  that  previous  mis- 
takes are  not  being  repeated,  and  that  existing  wheels  are  not  being 
reinvented. 

The  fundamental  issue  requiring  resolution  is  the  nature  of  SAT.  The 
original  concept  holds  that  training  program  development  cannot  be  procedur- 
alized,  and  that  it  is  therefore  the  proper  domain  of  the  expert  training 
program  designer.  The  second  SAT  concept  is  that  training  program  design  is 
procedural i Zable,  arid  that  inanuals  can  be  developed  which  are  usable  by  per- 
sonnel less  competent  and  less  costly  than  the  expert  to  produce  equally 
effective  training  programs.  The  resolution  of  this  issue  created  by  these 
two  conflicting  concepts,  probably  lies  in  their  synthesis.  The  fully  pro- 
cedural i zed  SAT  concept  has  no  basis  in  existing  psychological  theory  and 
research.  Yet  experience  with  it  has  shown  that  laymen  can  be  productive  in 
some  aspects  of  training  program  design.  The  original  SAT  concept,  which 
relies  on  the  expert,  provides  no  information  on  the  particular  skills 
necessary  to  qualify  someone  as  an  expert.  Hard  data  is  needed  to  determine 
what  skills  are  necessary  and  the  degree  of  proficiency  to  which  each  is 
required  for  the  accomplishment  of  the  various  steps  of  training  program 
development. 

The  original  concept  of  SAT  has  resulted  in  the  development  of  models 
of  these  steps,  but  the  question  of  general izability  is  left  to  the  expert. 
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The  procedural i zed  SAT  concept  has  also  resulted  in  models  (manuals)  but  with 
no  stated  limits  of  general izability.  The  former  is  not  desirable;  the 
latter  is  unacceptable.  Hard  data  is  needed  to  determine  which  aspects  of 
-training  proqram  development  can  be  procedural! zed,  the  degrv'je  to  which  they 
are  general izable.  When  these  questions  are  answered,  training  programs  can 
be  developed  to  bring  individuals  to  the  necessary  degree  of  competence  in 
the  skills  needed  to  use  the  mode’s  correctly.  The  ambiguities  within  each 
of  the  two  SAT  conceptualizations  must  be  resolved  in  order  to  allow  SAT 
research  proceed  more  rapidly. 

SEClIiK.  I - Organization  of  the  Bibliography 

Documents  falling  into  more  than  one  category  are  referenced  in  each  of 
the  relevant  sections.  Unauthored  military  documents  are  found  at  the  end  of 
each  section.  The  bibliography  has  been  divided  into  eighteen  sections,  each 
covering  a topic  important  to  training  program  deve iopment.  The  topics 
include  those  which  are  considered  by  the  proced jral ized  .AT  manuals,  such  as: 
task  analysis,  specific  behavioral  objectives,  sequencing,  media  selection, 
methodology  selection,  and  evaluation.  Also  included  are  other  topics  whicn 
must  be  considered  in  the  design  of  efficient  training  programs  but  which  are 
neglected  by  the  procedural i zed  SAT  manuals:  instructor  training,  instruc- 

tional management,  cost,  human  engineering,  simulation,  innovation  and  educa- 
tional technology.  The  remaining  sections  are:  ISD/SAT,  PI/CAI,  job 

analysis,  task  taxonomy  and  systems  analysis/operations  research.  A brief 
description  of  each  section  follows: 

SECTION  II  - Instructional  Systems  D'.velopment  (Systems  Approach  to  Training) 

The  general  re'^erences,  manuals,  articles,  etc.,  on  SAT  under  all  of 
its  r antes  (ISD,  DIS,  0.''T,  TSA,  SET.  SAY)  are  listed  together  with:  subject- 
matti  -specific  Ss"  manuals,  SAT  final  reports,  analytic  evaluations  of  the 
SAT  concept,  and  the  single  empirical  evaluation  of  a S.AT  manual.  All  SAT 
references  v/hich  could  be  found  are  included.  They  date  from  1951  to  1975. 
Perusing  this  section  will  aid  the  reader  in  obtaining  a perspective  on  the 
SAT  literature,  its  size,  its  chronology,  its  authorship  and  its  content. 

SECTION  III  - Evaluation 

Evaluation  permeates  every  aspect  of  training  program  design.  The 
training  technologist  is  called  upon  to  evaluate  schools,  programs,  teachers, 
students,  materials,  media,  and  concepts.  The  various  types  of  evaluation 
including  quality  control  and  the  factors  which  affect  them  are  referenced  in 
Section  III.  Although  statistics  is  an  integral  part  of  evaluation,  it  would 
have  been  unwieldy  to  include  the  entire  statistical  literature  in  this 
section.  Thus  only  a few  standard  texts  have  been  included.  Statistical 
questions  arising  during  training  program  design  which  are  not  covered  in  a 
basic  text  such  as  Hays  (1963)  should  be  referred  to  a competent  statistician. 

SECTION  IV  - Methodology  Selection 

Traditionally,  course  developers  have  relied  on  three  methods,  the 
lecture,  the  conventional  textbook  and  practice  with  the  operational  device, 
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for  all  of  trainino.  Although  such  proarans  are  usually  effective,  they  are 
probably  not  as  efficient  as  they  could  be.  One  focus  of  SAT  developmental 
efforts  has  been  to  increase  training  efficiency  by  tailoring  the  teaching 
methods  used  in  a course  to  the  subject  matter  and  to  the  specific  objectives 
of  the  course.  Although  most  of  the  procedural i zed  SAT  manuals  include  a 
"model",  that  is,  a set  of  rules  for  accomplishing  this,  none  have  been 
validated. 

SECTION  V - Media  Selection 

Efforts  to  increase  training  efficiency  have  also  included  the  selec- 
tion of  media  to  be  used  based  on  the  content  and  objectives  of  a course.  As 
with  methodology  sele«ftion,  the  ultimate  media  selection  device  would  be  a 
catalogue  which  specifies  the  optimal  media,  given  the  parameters  of  the 
training  task.  Although  most  procedural ized  SAT  manuals  have  included  such 
catalogues,  none  have  been  validated  and  none  have  been  widely  accepted. 
Sugarman,  Buckenmeier  and  Johnson  (1975)  state  that  such  catalogues  can  not 
be  maoe  workable  at  this  time,  because  the  necessary  information  is  not 
available.  Eckstrand  (1964)  DeGreen  (1970),  Braby  et  al,  (1975)  and  Monte- 
merlo  (1975)  arrived  at  the  same  conclusion. 

SECTION  VI  - Programmed  instruction  and  Computer  Assisted  Instruction 

Although  programmed  instruction  and  computer  assisted  instruction  are 
instructional  methods,  a separate  section  was  allocated  to  them  because  of 
the  vast  literature  they  have  produced.  The  most  comprehensive  bibliography 
on  programmed  instruction  and  computer  assisted  instruction  is  that  of  the 
Entelek  Corporation,  however,  since  it  is  a subscription  service  ana  may  not 
be  readily  available  to  the  reader,  Section  VI  is  included. 

SECTION  VII  - Task  Analysis 

There  is  universal  agreement  in  the  SAT  literature  that  task  analysis 
is  necessary.  However,  there  is  little  agreement  as  to  what  a task  analysis 
is.  No  satisfactory  ansv/ers  exist  to  the  following  questions.  Can  task 
analysis  be  procedural  ized?  If  so,  to  what  degree?  How  general  ir.able  would 
those  procedures  be?  What  skills  must  a good  task  analyst  have?  Hov?  can  one 
differentiate  between  a "good"  task  analysis  and  a poor  one?  If  two  well 
qualified  task  analysts  independently  analyze  the  same  task,  what  degree  of 
conmonal ity  would  their  output  have?  How  does  one  transition  from  task 
analytic  data  to  the  development  of  the  training  materials?  Section  VI  con- 
tains the  references  on  all  aspects  of  task  analysis  except  for  task  taxonomy 
(Section  IV)  and  job  analysis  (Section  VIII). 

SECTION  VIII  - Job  Analysis  - 

The  difference  between  job  and  task  analysis  is  a frequent  source  of 
confusion  among  newcomers  to  the  training  community.  The  two  differ  in  pur- 
pose, process,  and  product.  Job  analysis  is  a managerial  tool  for  allocating 
tasks  to  slots  (positions,  billets),  and  results  in  a job  description.  Such 
documents  contain  information  irrelevant  to  training  and  do  not  contain 
sufficient  information  on  which  to  base  a training  program.  Task  analysis 
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is  concerned  with  the  detailed  description  of  the  subtasks  of  a given  task 
area  with  the  heirarchical  relationships  among  them.  The  areas  of  job  and 
task  analysis  are  closely  intertwined  and  tne  reader  interested  in  either 
will  benefit  from  both  Sections  VII  and  VIII. 

SECTION  IX  - Task  Taxonomy 


Having  dete*'r’.ined  the  content  of  a training  program,  the  job  of  the 
course  desinner  is  to  oetermine  how  to  teach  it  most  efficiently.  In  other 
words,  methods,  media  and  sequences  which  are  most  appropriate  to  the  course 
content  must  be  chosen.  The  characteristics  used  to  describe  course  content 
are  known  as  the  "task  taxonomy  Eckstrand  (1964)  stated  that  until  a 
viable  task  taxonomy  which  relates  types  of  course  content  to  media  and 
methods  becomes  available,  course  design  will  be  more  of  an  art  than  a 
science.  To  date,  no  such  taxonomy  is  available.  In  spite  of  the  central 
role  of  task  ta.*OhOm>  in  training  technology,  very  little  research  effort  has 
been  expended  to  further  the  state-of-the-art.  The  high  risk  associated  with 
the  development  of  generally  applicable  taxonomies  has  caused  researchers  to 
concentrate  their  efforts  in  the  generation  of  taxonomies  specific  to  certain 
fields,  such  as  leadership  and  maintenance. 


SECTION  X - Specific  Behavioral  Objectives  (SBO's) 


The  state-of-the-art  of  SBO's  is  similar  to  that  of  task  analysis. 

There  is  widespread  agreement  that  SBO’s  are  important,  yet,  most  of  the 
important  questions  concerning  them  remain  unanswered.  For  instaiue.  althou^ 
writing  SBO's  adds  to  the  cost  of  course  design,  no  empirical  evidence  could 
be  found  concerning  the  circumstances  under  which  this  added  cost  is  counter- 
balanced by  savings  accrued  through  greater  training  efficiency.  Two  facts 
serve  to  emphasize  the  importance  of  answering  this'question.  One  is  that  a 
great  number  of  effective  training  programs  exist  for  which  explicit  SBO's 
were  never  written.  The  other  is  SBO  writers  often  beceme  ''bogged  down"  in 
detail.  Empirical  evidence  is  needed  concerning  the  point  past  which 
increased  detail  causes  more  problems  than  it  solves.  This  would  help  answer 
other  key  questions  such  as:  How  can  the  SBO  writer  dete*”mine  when  his  work 

is  "good  enough"  (that  is,  sufficiently  complete  and  in  a useful  form)?  Hew 


oaii  a uoiittact  monitor,  or  any  one  other  than  the  SBO  writer,  make  the  same 
determination?  What  training  and  experience  qualify  a person  to  write  SBO's? 
Given  two  qualified  SBO  writers  working  independently  on  the  same  course,  how 
different  will  their  outputs  be?  Will  it  make  any  difference  in  the  effi- 
ciency of  the  course  to  be  developed? 


SECTION  XI  - Sequencing 


The  training  program  designer  attempts  to  sequence  instructional  events 
to  obtain  maximal  transfer  of  training.  Perhaps  less  is  known  about  this 
process  than  about  any  other  stage  of  instructional  design.  The  most  obvious 
and  most  often  used  strategy  is  to  base  the  sequence  on  the  hierarchy  of  task 
subtasks  developed  during  task  analysis.  The  fallacies  inherent  in  this 
approach  and  the  reasons  why  it  should  not  be  adopted  as  a general  rule  are 
discussed  in  detail  by  Glaser  and  Resnick  (1972).  Unfortunately,  neither 
they  nor  anyone  else  provides  a viable  alternative  for  general  usage. 
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SECTION  XII  • Instructor  Training 

Procedural! zed  SAT  manuals  have  generally  paid  little  attention  to  the 
problems  of  how  to  select,  train  and  evaluate  instructors.  Yet,  the  most 
carefully  designed  training  program,  especially  an  innovative  program,  can 
not  succeed  without  the  support  of  the  instructors  who  will  implement  it. 

With  tlie  advent  of  modern  techniques  which  may  run  counter  to  the  experience 
and  beliefs  of  instructors,  the  adequate  preparation  of  instructors  for  these 
innovations  is  even  more  important.  Unlike  other  areas  of  instructional 
design  where  design  strategies  are  being  used  which  have  no  theoretical  or 
empirical  basis,  more  is  known  about  instructor  training  than  is  being  imple- 
mented. 

SECTION  XIII  - Educational  .Management 

The  careful  preparation  of  a training  program  provides  the  potential  for 
effective  and  ef^^icient  training.  The  degree  to  which  that  potential  is 
achieved  depends  on  how  the  program  is  managed. 

Careful  attention  must  be  paid  to  student  flow,  instructor  flow, 
scheduling,  maintenance,  supplies,  and  ancillary  services.  In  general,  the 
larger  the  instructional  program,  the  more  important  is  the  management  plan. 
For  instance,  a pilot  training  p»‘ogram  could  not  exist  without  a management 
plan,  while  the  effectiveness  of  a short  programmed  text  would  not  be  signif- 
icantly changed  if  it  was  not  accompanied  by  a management  plan.  The  lack  of 
attention  to  management  problems  by  the  procedural  SAT  manuals  indicates  that 
they  are  intended  only  for  developing  instructional  programs  of  the  latter 
type.  As  can  be  seen  in  Section  XIII,  interest  in  educational  management  is 
increasing. 

SECTION  XIV  - Cost 

The  primary  goal  of  training  designers  is  to  ensure  cost-effectiveness, 
that  is,  the  meeting  of  all  training  objectives  in  the  least  costly  manner. 
Yet,  the  determination  of  training  program  costs,  and  the  allocation  of 
those  costs  to  specific  portions  of  the  program  is  a difficult  task.  The 
generation  of  meaningful  cost  estimates  is  hampered  by  the  fragmentation  of 
monetary  responsibility,  the  variety  of  accounting  procedures  used,  and 
political  problems.  A further  deterrent  to  objective  determination  of  cost 
savings  is  the  pressure  which  is  often  brought  to  bear  on  SAT  teams  to  show 
cost  savings.  When  an  existing  training  program  is  re-developed  (ISD-ed, 
SA''^-ed)  its  objectives  often  change,  further  complicating  the  measurement  of 
relative  costs.  Section  XIV  can  provide  help  to  personnel  faced  with  such 
problems. 

SECTION  XV  - Innovation 

The  selection  of  appropriate  training  methods  and  media  can  have  side- 
effects  for  which  the  training  program  designer  should  be  prepared.  If  the 
selection  includes  methods  and/or  media  unfamiliar  to  the  instructors  and 
administrators  who  will  implement  the  new  training  program,  problems  associ 
ated  with  innovation  will  arise.  The  requirement  to  change  from 
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practices  wnich  are  familiar  and  comfortable  inevitably  arouses  anxieties. 
The  addition;/  factor  of  vested  interests  complicates  the  problem.  The 
innovative  course  designer  must  be  a skilled  politician  if  his  programs  are 
to  be  implemented,  and  more  importantly,  if  his  programs  will  be  continued 
after  his  involvement  ceases.  While  the  bibliography  in  Section  XV  does  not 
provide  solutions  to  all  of  the  problems  associated  with  innovation,  it  can 
make  the  reader  aware  of  many  of  those  problems  and  can  indicate  how  others 
have  attempted  to  solve  thc<ii  in  specific  instances. 


SECTION  XVI  - Educational  Technology 


The  professional  training  program  developer  is  called  on  to  make  judg- 
ments concerning  the  relative  efficacy  of  various  methods,  media,  sequences, 
etc.  To  do  this  he  draws  upon  his  experience,  nis  training  and  the  experi- 
ence of  others  as  described  in  the  professional  journals.  In  essence,  it  is 
this  collective  experience  which  defines  the  field  of  educational  technology. 


nology.  Section  XVI  of  this  bibliography  can  direct  the  reader  to  summaries 
of  the  field  published  up  to  now  and  can  lead  him  to  the  periodicals  which 
will  keep  him  up-to-date. 


SECTION  XVII  - Human  Engineering  and 


SECTION  XVIII  - Simulation 


The  training  program  developer  often  faces  decisions  concerning  the  use 
and  design  of  simulators  and  training  devices.  In  order  to  do  this,  he  must 
have  a working  knowledge  of  human  engineering,  the  state-of-the-art  of  sim- 
ulation hardware,  and  the  translation  of  training  requirements  into  hardware 
design.  The  majority  of  the  inputs  of  the  training  program  developer  will 
be  in  this  last  area.  Although  there  is  little  theory  available  to  help 
there  is  a wealth  of  empirical  data  based  on  past  experience  which  can  prove 
valuable.  Sections  XVII  and  XVIII  can  aid  in  locating  it. 

SECTION  XIX  - Systems  Analysis  and  Ope>"ations  Research 

The  field  of  "operations  research"  gave  rise  to  "systems  analysis"  which 
in  turn  led  to  the  original  conception  of  the  "systems  approach  to  training." 
The  final  step  in  this  evolution  has  been  the  procedural i zed  systems 
approach  to  training  which  is  also  known  as  instructional  systems  development 
(ISO).  A knowledge  of  the  precursors  of  ISO  can  greatly  increase  one's 
understanding  of  the  present  state-of-the-art.  Section  XIX  provides  the 
relevant  references. 
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